Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add payu dev environment #26

Merged
merged 10 commits into from
Jun 17, 2024
Merged

Add payu dev environment #26

merged 10 commits into from
Jun 17, 2024

Conversation

jo-basevi
Copy link
Collaborator

@jo-basevi jo-basevi commented Apr 10, 2024

Deploy a payu environment, with the latest changes on payu's main branch to pre-release location on gadi (e.g. /g/data/vk83/prerelease).

So environment location would be /g/data/vk83/prerelease/apps/payu/dev, and modulefile location would be /g/data/vk83/prerelease/modules/payu/dev.

The modulefile would be a symlink to the common modulefile in /g/data/vk83/modules/payu/.common, so have extended the common modulefile to check for fullpath of the modulefile being evaluated - to see if its in the prerelease folder.

Closes #22

Copy link
Member

@aidanheerdegen aidanheerdegen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks cool.

.github/workflows/deploy-dev.yml Outdated Show resolved Hide resolved
env-dev.yml Show resolved Hide resolved
modules/.common Outdated Show resolved Hide resolved
@aidanheerdegen
Copy link
Member

Would it make sense to use a containerised environment for this?

#14

If it slowed this PR down too much then it might make sense to make containerisation a subsequent PR.

@jo-basevi
Copy link
Collaborator Author

Would it make sense to use a containerised environment for this?

I've been wanting to learn about using singularity containers so I'll be keen to try that

Add checks if there has been any commits to payu's main branch since last successful workflow run - it was done this way rather than a CD workflow on payu-org/payu to prevent having to store repository secrets of payu-condaenv on payu-org/payu
@jo-basevi
Copy link
Collaborator Author

jo-basevi commented Jun 14, 2024

Updates:
Leaving containerised payu to a separate PR due to time constraints

Add a job to check if there's been any new commits to the master branch payu-org/payu repository, since the last successful payu dev deployment workflow run. This check will run periodically.
A new environment will get deployed if there's new commits, or if the environment file has changed or on manual dispatch (so can get new environment ASAP if needed).

New environment will get deployed to /g/data/vk83/prerelease/apps/payu/dev-$VERSION. Once installed the /g/data/vk83/prerelease/apps/payu/dev symlink will be created - these will be used for when the payu/dev module is loaded. The environment that the symlink previously pointed to be will remain, as the environment could be actively used at time of deployment. Older payu/dev environments are removed.

The versions of dev environment dev-YYYYMMDDTHHMMSSZ-$SHORT_PAYU_GIT_HASH, so they are unique. Having time included is a bit unwieldy, but I was thinking of the case where the dev environment file could be updated twice in one day - which probably wouldn't happen often..

Updated modulefiles deployment to generate modulefiles for release and prerelease and sync to respective locations (the common module file and env.sh uses different paths).

TODO:

  • Frequency/time of the cron job? Every hour/every day?

@jo-basevi jo-basevi marked this pull request as ready for review June 14, 2024 05:28
@CodeGat CodeGat self-requested a review June 14, 2024 05:38
CodeGat
CodeGat previously approved these changes Jun 14, 2024
Copy link
Contributor

@CodeGat CodeGat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Approving as the comments aren't necessary changes.

.github/workflows/deploy-dev.yml Outdated Show resolved Hide resolved
.github/workflows/deploy-dev.yml Show resolved Hide resolved
.github/workflows/deploy-dev.yml Outdated Show resolved Hide resolved
.github/workflows/deploy-dev.yml Outdated Show resolved Hide resolved
.github/workflows/deploy-dev.yml Show resolved Hide resolved
Copy link
Contributor

@CodeGat CodeGat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me :)

@jo-basevi jo-basevi merged commit bb1d164 into main Jun 17, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatically updated dev build
3 participants